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Abstract 


Past research on higher-order thinking (HOT) was mainly conducted on the bases of educational context 
in U.S. or western countries. This research aimed to see what kinds of HOT styles actually appear in 
universtiy students in South Korea. The use of HOT skills were explored in Korean universtiy students 
and the factors influencing the classification were examined. 1,138 Korean university students were called 
to respond to Lee s (2016) Higher-Order-Thinking-Scale for Korean University Students (HOTUS). Then, 
a latent profile analysis and the multinomial logistic analysis were conducted. The latent profile analysis 
revealed that the use of HOT skills could be classified into four classes (i.e., a lower-order thinking 
class, a creative-argumentative class, an analytical-caring class, and a higher-order thinking class). 
Gender, year, and instructional approach were the determinants of latent profile types. However, there 
were no differences when measured by academic fields. Students with lower years were likely to fall under 
lower-order thinking class. The probability that men was classified as a caring class was statistically 
significantly lower than that of women. Students who received lecturer-centered learning were more likely 
to fall under the analytical and caring class. 

Keywords: higher-order thinking skill, latent profile analysis, multinomial logistic analysis. 


Introduction 


Today, it is not difficult to find a college or university without courses on creativity, fusion, 
and personality. Many higher educational institutions cite HOT skills as the core capabilities 
students need to develop during their university life and make constant efforts to foster those 
skills (Klein, Benjamin, Shavelson, & Bolus, 2007). Higher-order thinking (HOT) skills, such 
as creativity, have become an important quality required of learners in 21st century educational 
environment (Retnawati, Djidu, Kartianom, Apino, & Anazifa, 2018), and accordingly, interest 
in HOT as the core skills of students is increasingly growing (Murphy, Bianchi, McCullagh, & 
Kerr, 2013). 

HOT skills discussed in the context of educational psychology goes quite far back. Many 
scholars have encouraged the development of HOT skills as an outcome of learning and teaching 
(Vygotsky, 1978), and emphasized education that requires higher levels of learning through 
analysis, synthesis and evaluation, rather than simple acquisition of knowledge (Bloom, 1956). 

HOT skills mean cognitive processes at high levels, and at the same time they include 
emotional characteristics involving in-depth contemplation of problem situations, provision 
of appropriate grounds for problem solving and generation of new approaches from multiple 
perspectives while considering others (Lee, 2016). These HOT skills are consistent with the 
creativity, communication, collaboration, and critical thinking which are advocated as the 
key skills for the 21st century (Soulé & Warrick, 2015). HOT skills involve critical thinking, 
creative thinking, and caring thinking (Lipman, 1995) and active participation is essential in 
such learning scenes (Woolever & Scott, 1988). Use of HOT skills, in most cases, accompanies 
kinds of activities requiring a stance of argumentation (Alexander, et al., 2011). 
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Problems of Research 


Recently, many universities in Korean society emphasize HOT as core skills in preparation 
for environmental changes in 21‘ century. Regardless of the long history of the HOT academy 
in western culture, little studies were conducted to investigate what kinds of HOT styles 
actually appear in university students in countries other than U.S. This research was conducted 
in hope of finding the profiles existing behind the HOT skills of Korean university students and 
to understand how the profiles obtained from the analysis differ according to different years, 
genders and academic fields. Based upon the research problems, we addressed the following 
two research questions: 

What do the HOT skill profiles of Korean university students look like? 

How do the HOT skill profiles of Korean university students differ according to different 
years, genders, academic fields and instructional approaches? 


Research Focus 
Higher-Order Thinking Skills 


In its long history, the concept of thinking or thought has changed in diverse ways. 
Thinking can be simply "everything that comes to mind or goes through the head" but it is also 
"seeking the ground or basis for one's belief and its adequacy to support the belief" (Dewey, 
1999). Dewey called the latter concept a "reflected thought," which alone, he believed, was 
truly educative in value. Drawing upon this concept, philosophers drew the concept of critical 
thinking and reflective thinking and psychologists defined the concept of problem solving skills 
(Lewis & Smith, 1993). In the late 1980s, these concepts together began to be emphasized 
in social studies under the name "higher-order thinking skills" (Jeon, 2010) in Korea. These 
higher-order thinking skills refer to "a cognitive ability to solve a problem critically, creatively 
and caringly when individuals encounter a problematic situation, instead of dealing with it 
with familiar knowledge or experience" (Lipman, 1995; Newmann, 1987). Given the existent 
literature, we saw Higher-order thinking skills as comprising components of creative thinking 
skills, critical thinking skills and caring thinking skills. 

Creative thinking skills refer to the ability in the cognitive domain of creativity 
(O’Sullivan & Guilford, 1975) which allows people to think up new and useful ideas when 
they face problem situations (Amabile et al., 2005; O'sullivan & Guilford, 1975). Creative- 
thinking has the dimensions of fluency, flexibility, originality and elaboration. Fluency is the 
ability to generate a large number of ideas and flexibility is the ability to shift approaches and 
find solutions from new and unconventional perspectives. Originality is the ability to generate 
original and unusual ideas and elaboration is the amount of detail or completeness manifested 
to existing ideas (Torrance, 1988) 

Critical thinking refers to the cognitive domain of critical thinking (Facione, 1990) which 
is the ability to make accurate and sound judgments based on the given criteria when people 
are faced with certain problems (Bandman, 1988; Kemp, 1963; Norris, 1985). It is analytical, 
argumentative and dialectical. Analytical thinking means "dividing one whole into many parts, 
examining in detail, thoroughly investigating issues or situations" and argumentative thinking 
involves "drawing conclusions from given data or providing appropriate grounds to justify 
certain argument." Dialectical thinking is "the ability to connect many parts into one whole 
or the ability to approach given situations with different ideas or the ability to generate better 
alternatives from broader perspectives" (M. Kim, 2007, pp. 25-34). 

Caring thinking refers to the cognitive domain of caring (Soo-dong Kim, Ahn, & Lee, 
2014) which involves a cognitive ability (Jin-sook, 2011; Park, 2011)to allow people to make 
sound judgments when they encounter certain problems based on interest and empathy of others. 
Lipman sees caring thinking as having five features: appreciative thinking, affective thinking, 
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active thinking, normative thinking and empathetic thinking, and these features have 39 sub- 
features. Appreciative thinking is "to pay attention to what matters, to what is of importance," 
and affective thinking is "dealing with others with love, joy, friendship, sadness, anger and 
other emotional attachments." Active thinking is "to lead selfless actions to actively take care 
of others," normative thinking is "to think, reflect and explore one's own values required in 
society," and empathetic thinking is "to feel the emotions and experiences of others in others' 
positions" (Lipman, 2003, pp. 264-270). 


Measurement of Higher-Order Thinking Skills 


The scale employed in this research is HOTUS (Higher-Order Thinking Skill Scale for 
University Students) developed by Lee (2016). Developed to meet the special conditions of 
Korean university students' academic scenes, the scale has the following characteristics: 

HOT is classified between a cognitive domain and an affective domain and an inventory 
has been developed by characterizing the cognitive domain. HOT has been studied without a 
clear distinction between cognitive domain and affective domain, but a closer look into these 
previous studies reveals that they mostly focused on the cognitive domain. As shown in Figure 
1, this research has a clear distinction between the domains and the elements under HOT are 
also composed as the elements of the cognitive domain. 


Higher-order thinking skills 
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Figure 1. Structure of higher-order thinking. 


This scale reflects all HOT skills derived from the early development of precedent study 
and consists of a total of 133 question items across 8 categories of factors: 14 items in fluency, 
11 items in flexibility, 14 items in originality, 15 items in elaboration, 15 items in analysis, 
15 items in argumentation, 23 items in dialectical and 26 items in caring. The scale was then 
reviewed through an expert meeting, university student pilot study and validity and reliability 
check and eventually narrowed down to form 25 items in 5 factors. The final selection of factors 
and relevant items is as shown in Table 1. 
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Table 1. Attribute definitions and the number of items by sub-factor of higher- 
order thinking skills for Korean university students. 





Sub-Factor 





(number of items) Definition 

Analysis (4) Cognitive ability to contemplate a problem situation in detail 

Creativity (4) Cognitive ability to generate new and useful ideas when facing problem situations 

Argumentation (5) Cognitive ability to provide adequate grounds for justifying conclusions about problem 
situations 

ree Cognitive ability to provide, synthesize and devise new forms of alternatives from 

Dialectic (5) ; . re 
various perspectives about problem situations 

Caring (7) Cognitive ability to seek reasonable approaches or thoughts about problem situations 


based on interest and empathy of others 
Note. The table is adopted from Lee (2016). 





Determinants of Use of Higher-Order Thinking 


According to the meta-analysis published by Budsankom et al. (2015), learning 
environment and psychological cognitive characteristics of students strongly and directly 
predicted HOT skills (96.8% variance explained). Family variable was also a determinant of 
HOT skills but its indirect effect was bigger. In that study, Budsankom et al. collected HOT 
skill-related studies carried out in Thailand from 1999 to 2013 and analyzed the mathematical 
structural equation modeling (MASEM). The analysis of 166 studies revealed that the family 
variable did not have a significant impact overall, but the learning environment or personal 
variables had a substantial impact. In particular, while the psychological characteristics of 
each student were also a direct determinant of HOT skills, they played a role as a medium for 
the school environment and family variables to influence HOT skills. In addition, the school 
variable was found to have a rather indirect impact, because the school variable, including the 
way a Class is run or managed, contributed to facilitating HOT skills by fostering the attitudes, 
feelings, knowledge and other psychological aspects of students (Schumacker & Lomax, 2010). 

In fact, many studies in the past reported that instructional approaches and free academic 
activities in universities contributed to enhancing critical thinking, creative thinking and other 
high-order thinking skills. For example, the learning models that focus on learners, such as 
problem-based learning, are known to be a classic learning method to enhance HOT skills 
in students (Raiyn & Tilchin, 2015). Learning mathematics through contextual class, rather 
than through symbol or text-based learning environment is also known to have contributed to 
improving HOT skills (Samo, Darhim, & Kartasasmita, 2017). According to the recent meta- 
analysis, interactive class, practical and contextual problem-based class and teachers' mentoring 
were the factors that affected the critical thinking ability (Abrami et al., 2014). Student-centered 
learning and teaching method such as problem based learning (Mokhtar, Tarmizi, Tarmizi, & 
Ayub, 2010) and project based learning (Vidergor & Krupnik-Gottlieb, 2015) have been called 
to develop HOTS. 


Methodology of Research 
General Background of Research 
Given the precedent research, this research focused on exploring the types of HOT 
utilization and the factors that determine such utilization of HOT. As candidate variables that 


influence the formation of HOT, school and student variables, examined by Budsankom et al. 
were included. The school variables included in this research are the ways a classroom is run 
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(Lecturer-centered, Lecturer and student-centered, student-centered) and the student variables 
included gender, academic fields and years. Especially, the general expectation that students' 
ability to utilize HOT would increase during university life was employed. In addition, it was 
expected that there would be differences in the way HOT is utilized among students, based on 
the thinking pattern theory according to gender and academic fields (e.g., Onyeruru, 2015; Seo, 
2003). 


Sample of Research 


A survey was conducted in November 2015 for university students from 6 regions 
(Seoul, Gyeonggi-do, Gangwon-do, Chungcheong-do, Jeolla-do, and Gyeongsang-do). A total 
of 1,713 copies of the questionnaire were distributed, of which 1,226 copies were collected. Of 
these, 88 copies with unreliable responses or missing values were removed, and 1,138 copies 
were eventually analyzed. All the subjects agreed to participate in the research by signing a 
written consent form. A higher percentage of lower year students participated in the survey. 
Specifically, 501 students (44.02%) were male and 506 students (44.46%) were in year 1, 265 
students (23.29%) in year 2 and 230 students (20.21%) in year 3. Table 2 shows the demographic 
characteristics of the research participants. 


Table 2. Demographic characteristics of study participants. 

































































Variable N % 
ees Male 501 44.02 
Female 637 55.98 
Year 1 506 44.46 
a Year 2 265 23.29 
Year 3 230 20.21 
Year 4 137 12.04 
Humanities and Social Sciences 640 56.24 
Science and Engineering 260 22.85 
Academic Field Arts and Physical Education 162 14.24 
Medical 58 5.10 
Others 18 1.58 
Seoul 259 22.76 
Gyeonggi-do 342 30.05 
me Gangwon-do 110 9.67 
Chungcheong-do 176 15.47 
Jeolla-do 152 13.36 
Gyeongsang-do 99 8.70 
Lecturer-centered 885 778 
Instructional Lecturer + students 167 14.7 
Approach Student-centered 86 7.6 
Total 1138 100.0 
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Instrument and Procedures 


HOTUS (Higher-Order Thinking Skill Scale for University Students) consists of the sub- 
categories of creative thinking, critical thinking and caring thinking, and each sub-category 
includes question items regarding creativity, analysis, argumentation, dialectic and caring that 
explain HOT skills (Lee, 2016). Creativity is the cognitive ability to generate new and useful 
ideas; analysis is the cognitive ability to examine problem situations in detail; argumentation 
is the cognitive ability to give an appropriate basis for the expected conclusion; dialectic is the 
cognitive ability to put forward and synthesize various perspectives and create new alternatives; 
and caring is the cognitive ability to seek a reasonable method or idea based on interest and 
empathy of others. HOTUS contains a total of 25 questions (4 questions for creativity, 4 
questions for analysis, 5 questions for argumentation, 5 questions for dialectic and 7 questions 
for caring) and averages of scores on a 5-point Likert scale (1 = very unlikely, 5 = very likely) 
are calculated. The internal consistency reliability of HOTUS was reported as .879 (Lee, 2016). 
The definitions and the number of questions of each subconstruct in this scale are as shown in 
Table |. The detailed list of questions is provided in the Appendix. 

Variables (gender, year, academic field, location and instructional approach) to be treated 
as independent variables were also surveyed. Students were asked to choose one academic 
field from © humanities and social sciences @ science and engineering, @ arts and physical 
education © medical and © others. As to instructional approach provided to students, students 
chose one of the following five options: © more than 90% of Lecturer-centered ® 70% of 
Lecturer-centered + 30% of student participation @ 50% of Lecturer-centered + 50% of student 
participation, © 30% of Lecturer-centered + 70% of student participation, and © more than 
90% of student participation. For the convenience of analysis, © and @ replies were grouped as 
Lecturer-centered, © replies were treated as Lecturer + student style, and ®@ and © replies were 
grouped as student-centered. 


Data Analysis 


In this research, latent profiles of HOT skill types were classified and the profiles were 
then analyzed using M plus 7.11 to validate the influence of the relevant variables. In class 
classification, the sub-classes estimated from the analysis of replies are called latent profiles. 
The latent profile analysis is called a person-oriented approach that distinguishes classes 
through personal characteristics and attributes (Bergman & Magnusson, 1997). The latent 
profile analysis relies on data to produce the number of classes, and does not require statistical 
assumptions (linearity, normality of data, homogeneity of variance) required in cluster group 
analysis (Sa-hyeon Kim & Hong, 2010). The ability to overcome the limitation that three or 
more nominal scales cannot be analyzed is another merit of this analysis (Yang, Kim, & Hwang, 
2007). 

First, entropy was checked to identify the quality of the classification. Entropy is between 
0 and 1, and its value increases when the probability of belonging to one latent class is closer 
to 1 and the probability of belonging to another latent class is closer to zero. Entropy with 
approximately 0.8 or more can be seen as a good classification (Clark, 2010). Next, the fit of the 
model-data, such as AIC (Akaike Information Criterion), BIC (Bayesian Information Criterion), 
and SABIC (Sample-size Adjusted BIC) was verified. Finally, comparison and validation for 
the model was conducted using the Mendell-Rubin adjusted likelihood ratio test (LMRLRT) 
and the parametric bootstrapped likelihood ratio test (BLRT). 

Multinomial logistic analysis was employed to identify candidate variables (gender, 
academic field and instructional approach) that influenced the formation of the latent profiles. 
In performing this analysis, the latent profiles were input as dependent variables and other 
influencing factors were input as independent variables. As the multinomial logistic analysis 
should be interpreted on the basis of the reference group, the last group set as the default value 
in SPSS was used. 
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Figure 2 shows the research model of this research. The model was intended to find latent 
profiles from the 25 questions of HOT skills and to explore the degree of impact of the variables 
-i.e., gender, academic field, location and instructional approach - on the formation of the latent 
profiles. 


Analysis1 





i 


Analysis4 


Creativity 1 





Creativity 4 
Gender, 
Year, 


: Argumentation 1 
Academic 


Latent 


Field, 
Instructional 
Approach 


profile 


Argumentation 5 





Dialectic 1 





Dialectic 5 


Caring 1 





Caring 5 


Figure 2. The Research Model of HOT LCA. 


Results of Research 
Latent Class Analysis 


Table 3 summarizes the descriptive statistics of the sub-variables of HOT skills employed 
in the latent class analysis. As seen in the Table, the five sub-variables of HOT skills showed 
a statistically significant correlation of an approximately medium level, but in some matching 
pairs such as between analysis and creativity and between caring and creativity, the correlation 
was very low or hardly existent. 
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490| Table 3. Descriptive statistics and correlations among sub-variables of higher- 


order thinking skills. 





























l 1 2 3 4 5 
1.Analysis 1 

2.Creativity ea 1 

3.Argumentation 501" .268* 1 

4.Dialect .408** 449** 428** 1 

5.Caring .385"* .070* .396** .378** 1 
Mean 3.72 2.92 3.50 3.36 3.74 
S.D. 0.61 0.68 0.56 0.59 0.54 





Notes. *p<.05, **p<.01 


Table 4 shows the results of the latent profile analysis for HOT skills of university 
students. In the latent profile analysis, the number of classes exposed from the data is determined 
considering the ease of interpretation and the fit of the model. The fit of the model was checked 
by increasing the number of profiles from two to five. 


Table 4. Model fit by the number of latent profile classes. 

















AIC BIC SABIC Entropy LMRLRT BLRT 
2 Profiles 64380.764 64763.578 64522.179 0.876 0.00 0.00 
3 Profiles 63314.271 63828.047 63504.065 0.844 0.00 0.00 
4 Profiles 62586.207 63230.946 62824.38 0.841 0.01 0.00 
5 Profiles 62119.997 62895.699 62406.55 0.862 0.22 0.00 








It is known that the comparison results of model fits are well differentiated when the 
entropy index is 0.8 or higher (Muthen & Muthen, 2007). The classification was almost 
accurate with the entropy values of the models compared in this research all showing 0.8 or 
more. Values of AIC, BIC, and SABIC all decreased as the number of latent profiles increased. 
LMRLRT was significant for up to four models, and BLRT was significant for all models, 
so it is difficult to determine which model is fit with these criteria. A researcher's judgment 
becomes important when multiple indices show different results (Kim Sa-hyeon et al., 2010), 
and in such cases, it is necessary to comprehensively consider various fit indices and theoretical 
interpretability (Cho Jee-yong et al., 2013). A four-class classification was employed in this 
research, comprehensively taking into account the ease of interpretation, fit index and other 
aspects. 
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Higher-Order Thinking Skills LCA 
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Figure 3. Means of item scores per LCA class of higher-order thinking skills. 


Figure 3 outlines the latent profiles produced. The characteristics of each type of 
profile are as follows: First, Class | showed low scores in most indicators including analysis, 
argumentation, dialectic and caring compared with other groups. This class can be termed "a 
lower-order thinking class." 169 students belonged to this class, accounting for 15% of all 
students. Class 2 earned relatively high scores, ranking second in creativity, argumentation and 
dialectic among all classes, but its caring score was relatively low. This class can be named 
"a creative and argumentative class." 330 students, or 29% were in this class. Class 3 had 
high scores in analysis, argumentation and caring and relatively low scores in creativity and 
dialectic. Given the high construct correlation between creativity and argumentation among 
the lower HOTUS scores, this group can be understood as having weak creative and divergent 
thinking but having exceptionally strong analytical and caring mind. Therefore, it can be called 
"an analytical and caring class." This class had 290 students, which comprised 25% of the total 
number of students. The last class is the so-called "higher-order thinking class" whose scores 
were high across all subconstructs. 349 students belonged to this class, accounting for 31% of 
the total number of students. 


Factors That Influence the Latent Profiles of Higher-Order Thinking Skills 


Table 5 shows the basic descriptive statistics of the four classes according to the variables 
of interest of this research. Cross-tabulation analysis produced a statistically significant 
difference overall, but no significant difference was produced only in academic fields. 

In gender indicator, the ratio of female students in the analytical and caring class was 
much higher than that of male students. According to Chi-square analysis, such difference was 


2 
statistically significant (4 = 24.851, p = .001). 198 students belonged to the analytical and 
caring class, accounting for 31% of 637 participating female students. On the other hand, the 
number of male students in the same class was 92, or only 18% of 501 students. 


2 

HOT types were significantly different by year (4 = 23.594, p = .005). Overall, the 
percentage of HOT students increased in year 4. Among the first-year students, 145 students 
belonged to the HOT class, accounting for only 29%, but in year 4, 50 out of 137 fourth year 
students, or 36% belonged to the HOT class. The proportion of students in the analytical and 
caring class also increased as the year level increased. Conversely, the percentage of students 
belonging to the lower-order thinking class decreased as the year level increased (20% in year | 
vs. 40% in year 4). The creative and argumentative class showed a high ratio in year 2 at 34%, 
but the ratio declined in year 4. 
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In academic fields, the most prominent among science and engineering students were 
the HOT class and the creative and argumentative class which accounted for 31% of student 
subjects, respectively. Students in arts and physical education were most populated in the 
creative and argumentative class (31%), while many of the medical students belonged to the 
high-order thinking class (41%). Among the students in humanities and social sciences, the 
biggest number of students was in the HOT class (30%), followed by the analytical and caring 
class (29%). However, the difference was marginal, indicating that there was no statistically 


2 
significant correlation between academic fields and HOT skill patterns ( x = 18.440, p =.103). 


Table 5. Four-Class composition based on variables of interest (N, %). 





Lower-Order Creative -Ar- 


















































5. ree : Analytical - Higher-Order Pearson 
Variable Thinking gumentative : aes 
Clase Gloss Caring Class ThinkingClass (p-value) 
Total (N=1138) 169 15% 330 29% 290 25% 349 31% 
Male (N=501) 87 17 155 31-92 18 167 33 24.851 
Gender (.001) 
Female (N=637) 82 13 175 27 198 31 182 29 
Year 1 (N=506) 101 20 133 26 127 25 145 29 
¥ Year 2 (N=265) 27 10 90 34 63 24 85 32 23.594 
ear 
Year 3 (N=230) 27 12 «71 31 63 27. «69 30 (.005) 
Year 4 (N=137) 14 10 36 26 37 27 50 36 
Humanities and 
Social Sciences 82 13 177 28 183 29 193 30 
(N=640) 
Science and Engi- , 
Aca- 
ome neering (N=260) 44 7 80 31 55 21 81 31 18.440 
: Arts and Physical (.103) 
Field Education (N=162) 34 20 52 31 37 22 44 26 
Medical (N=58) 6 10 15 26 13 22 24 41 
Others (N=18) 3 17 6 33 2 11 7 39 
Lecturer-centered , 
Instruc- _(N=886) 132 5 250 28 241 27 262 30 
tional Lecturer + stu- ; 14.223 
Ap- dents (N=167) 26 6 50 30 40 24 51 31 (.027) 
proach _ Student-centered 11 13 30 35 9 10 36 42 


(N=86) 





2 

The differences by instructional approach were also significant (4 = 14.223, p = .027). 
Among the students who received Lecturer-centered learning, the percentage of HOT class was 
slightly higher (30%) than the other classes (lower-order thinking, creative and argumentative, 
analytical and caring). This figure is small compared with 42% of HOT class students among 
the students who received student-centered learning. 

To identify the factors that influence latent profiles, a multinomial logistic analysis 
was conducted using the higher-order thinking class (Class 4) as a reference group. Among 
the candidate influence factors, gender, year, and instructional approach were found to be 
statistically significant, which is consistent with the statistics in Table 5 (the last line of Table 6). 
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Table 6. Multinomial logistic analysis results of higher-order thinking skill types. 





































































































Odds Ratio Standard Error Dp 
Male 1.056 .200 .186 
Year 1 2.631 336 .004* 
Year 2 1.276 .386 527 
Year 3 1.468 381 314 
1. Lower-Order Humanities and Social Sciences 936 125 .928 
Thing Gees Science and Engineering 1.261 144 158 
Arts and Physical Education 1.664 145 494 
Medical 597 841 540 
Lecturer -Centered 1.702 376 158 
Lecturer + student 1.736 437 207 
Male 943 163 120 
Year 1 1.298 254 805 
Year 2 1.528 215 123 
5 Year 3 1.447 279 185 
Creative and Humanities and Social Sciences 1.016 580 .978 
Argumentative - —— 
Class Science and Engineering 1.188 .598 174 
Arts and Physical Education 1.363 .603 .608 
Medical 122 653 618 
Lecturer -Centered 1.138 271 633 
Lecturer + student 1.170 326 .630 
Male 488 175 <.0001 
Year 1 1.072 207 187 
Year 2 805 287 451 
Year 3 1.153 285 616 
3. Analytical and Humanities and Social Sciences 2.662 831 .239 
Caring Class ~ Science and Engineering 2.261 849 337 
Arts and Physical Education 2.512 854 281 
Medical 1.428 893 .690 
Lecturer -Centered 3.352 391 .002* 
Lecturer + student 2.777 437 .020* 
Model Fit -2 Log Likelihood 651.823 
Chi Square 79.955 SIgninganee RIOD. 2.0004 
ability 
Note. * p<.05 


Specifically, in the lower-order thinking class, the first year showed a significant 
difference compared to the fourth year (p=.004), which is the reference group. Among the first 
year students, the odds ratio classified as a lower-order thinking class was 2.631, which is 
163.1% higher than fourth year students. This tendency was also very high in the second year 
students (27.6%) and in the third year students (46.9%), but statistical significance was not 
found (p=.527, p=.314, respectively). Gender, academic field and instructional approach did 
not have a big impact on the classification of students as a lower-order thinking class compared 
with the higher-order thinking class. 
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All the variables related to the creative and argumentative class did not appear to act as 
a cut-off point. The significance probability of each variable for the creative and argumentative 
class was all less than the significance level of .5 

The probability of belonging to the analytical and caring class seemed to be more obvious 
than the reference group (higher-order thinking class) according to gender and instructional 
approach. The probability that men will be classified as a caring class was statistically 
significantly lower than that of women. In case of men, the odds ratio of belonging to the 
analytical and caring class was 0.488, which is interpreted to be significantly reduced by 51.2% 
[(1-0.48)*100] compared to that of women (p<.001). Students who received Lecturer-centered 
learning showed that the odds ratio of belonging to the analytical and caring class increased by 
235.2% compared to other students who received student-centered learning. On the other hand, 
the odds ratio of belonging to the analytical and caring class among the students who received 
Lecturer + student type of learning increased by 177.7% compared to those who received 
student-centered learning. 


Discussion 


In this research, the possibility of classifying the use of HOT skills was explored and if 
such classification was possible, the factors influencing the classification were examined. To 
do this, a survey was conducted for Korean university students using the inventory for HOT 
skill utilization developed by Lee (2016), and the survey data was used to perform latent profile 
analysis and the multinomial logistic analysis. 

The latent profile analysis revealed that the use of HOT skills could be classified into a 
total of four classes. Various indices used for the classification clarified the distinction of the 
classes. Based on the characteristics of these four classes, they were named as a lower-order 
thinking class, a creative and argumentative class, an analytical and caring class, and a higher- 
order thinking class, respectively. The lower-order thinking class showed very low scores in all 
of the five subconstructs (analysis, creativity, argumentation, dialectic, and caring) of higher- 
order thinking skill utilization. On the other hand, the higher-order thinking class showed high 
scores in all subconstructs. 

The creative and argumentative class and the analytical and caring class have very 
unique characteristics. The creative and argumentative class showed high scores in creativity 
and argumentation on average but showed average or lower than average scores in other areas. 
In other words, students in creative and argumentative class act creatively when they solve 
problems in an original way or actively participate in discussions, but they are somewhat timid 
about following rules or caring or empathizing with others. In the analytical and caring class, 
the creativity score distribution was very low, but the analysis and argumentation scores were 
high, and the caring was significantly higher than the average. The analytical and caring class 
is a group of students who seem to have HOT ability but somehow lack flexibility or creativity. 
This type of students is common in real-life schools. 

To understand the background of students from each of the four classes identified 
from the latent profile analysis, gender, year, academic field, and instructional approach were 
examined and the differences were compared. The results were generally consistent with the 
findings of the precedent study. That is, gender, year, and instructional approach were the 
determinants of latent profile types (e.g., Schumacker & Lomax, 2010). However, unlike the 
previous discussions on different learning style across domains (Seo, 2003), there were no 
significant differences when compared by academic fields. 

Among many factors, the differences in instructional approaches and the differences 
in the development of HOT skills have already been strongly asserted in previous studies. In 
other words, the assertion of previous studies (Abrami et al., 2014) that contextual learning 
focusing on students such as PBL and PjJBL (Mokhtar et al., 2010; Raiyn & Tilchin, 2015) 
would help increase HOT skills in students was somewhat proved through this research. In 
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particular, it was found that students who received Lecturer-centered learning were more likely 
to fall under the analytical- caring class than creative-argumentative class. Considering that the 
students who were classified as the analytical and caring class in this research are more likely to 
display lower than average scores in creativity and dialectic, this phenomenon seems plausible. 
In other words, it can be interpreted as meaning that the students who took learner-centered 
class are more likely to be creative and flexible and put forth opinions from different point of 
views. On the other hand, the more Lecturer’s guidance is involved, students are more likely 
to think attentively and care for others, although they are less creative. While it is encouraging 
in that this type of thinking is consistent with what learner-focused classrooms intended to 
achieve, whether only learners' initiative should be emphasized in classrooms is questionable, 
considering that the analytical and caring aspects cannot be ignored in this research's base 
model for HOT skills (Lipman, 2003). As much as the need to be creative and self-assertive 
depending on situations and society, one needs to be mindful and receptive to others and in 
this regard, the argument that all university curriculum or instructional approaches should be 
centered around learners needs further discussions. 

Differences in the use of HOT skills by academic field were not statistically significant 
and this is primarily related to the background against which this inventory was developed. One 
of the objectives of the inventory development intended by Lee (2016) was to make a cross- 
disciplinary and cross-sectional inventory that could be used irrespective of academic fields. 
This intention was confirmed again by the latent profile analysis, thus enhancing the validity of 
the development and use of the inventory. 

This research also identified the differences in the use of HOT skills according to the year 
level, which is a natural and encouraging finding in the larger context of promoting HOT skills 
through university education. However, whether these differences are due to the actual changes 
occurring to individual students with year change or whether they are the result of learning in 
universities left room for further analysis through development-specific research design. 


Conclusions 


The current research findings indicate that different HOT styles appear in Korean 
University students. A considerable number of students appeared to contain high levels of HOT 
skills in every underlying construct (i.e., Analysis, Creativity, Argumentation, Dialect, and 
Caring), which was named ‘higher-order thinking class’. A small portion of the students were 
likely to show less usage of the constructs (lower-order thinking class). In the middle, there 
were two classes of students with greater use of creativity and argumentation and those with 
greater use of analysis and caring. The different HOT styles in the students were significantly 
related to their gender, year levels, and instructional method they have taken. 

The different revelation of the HOT styles and the significant relation to the learner 
variables call us to think appropriate teaching method for University students for 21st century. 
Many schools in the present era tend to incorporate student-centered learning approach 
regardless of their school philosophy merely due to the current educational trend. Different 
HOT styles seemed to appear along with the students’ learning environments. It seems necessary 
to see what kinds of higher-order thinkers the school desires to produce and then to set out 
appropriate instructional method for the students. The current study particularly found creative- 
argumentative graduates are likely to come forth in greater numbers when student-centered 
teaching was provided, whereas analytical-caring graduates are more produced with Lecturer- 
centered teaching. Students’ talents should be considered instead of simply striving to increase 
all the underlying HOT constructs. 
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498 Appendix 
Higher-Order Thinking Skill Scale for Korean University Students (HOTUS) by 
Lee (2016) 

No. Item Factor 

1 | review what | studied in class to solve a task. Analysis 
When | study, | try to interpret the meanings contained in learning materials such as F 

2 : ; Analysis 
graphics and pictures. 

3 | don't just memorize what | am going to learn, but | rather think about it carefully first. Analysis 

4 When | get stuck while studying, | guess the topic in the context. Analysis 

5 When | solve a task, | look for ways other than the way | have always used. Creativity 

6 | make novel ideas from different sources which my teacher did not suggest. Creativity 

7 | try to give more unique opinions than other friends. Creativity 

8 When | solve a task, | use different methods from others. Creativity 

9 | look for various grounds to solve a task. Argumentation 

10 When | solve a task, | draw conclusions based on actual data. Argumentation 

"1 When | collect learning materials, | look for errors based on accurate evidence. Argumentation 

12 When | solve a task, | tend to draw conclusions based on various data. Argumentation 

13 _| decide on an answer based on rational judgment and provide evidence. Argumentation 

14 __| present many opinions when | discuss with other friends. Dialectic 

15 When! discuss, | put together different opinions and then present a new opinion. Dialectic 

, | tend to induce my friends to change their viewpoints so that we can agree on one Dialectic 
opinion. 

17 | try to put different issues together and suggest ways to help solve the problems directly. Dialectic 

18 | examine pros and cons of other people's opinions and accept them as one opinion. Dialectic 

19 _| actively gather and accept opinions of others. Caring 

20 _| make a decision after listening to others. Caring 

21 | do not ignore other people's different opinions when | make a decision. Caring 

) If my friend tells me that my way of solving a task is wrong, | willingly accept that opinion Caring 


and try to change the solution. 
23 | explain to others who have different opinions so that they don't feel offended. Caring 
When | work on a team assignment, | am always mindful of keeping the given rules so as 




















a4 not to interfere with the team. Calg 
25 When | disagree with other friends, | try to understand them. Caring 
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